Problems with the Brainweb MRI Simulator in the Evaluation of Medical Image Segmentation Algorithms, and an Alternative Methodology

We demonstrate that simulated MR images obtained from Brainweb do not model the partial volume effect in a realistic fashion, and therefore cannot be used to evaluate medical image segmentation algorithms that rely on models of intensity distributions and incorporate partial volume effects. However, we make two observations; first, evaluation of segmentation algorithms on simulated data can only prove consistency between the assumptions incorporated into the simulation and segmentation algorithms, and second, given this constraint, a method for producing approximately noise-free MR images is all that is required to draw any conclusions that could be drawn through the use of Brainweb simulated images. We use these observations to motivate an alternative method for evaluating medical image segmentation algorithms, based on the use of the multi-dimensional segmentation algorithm provided by TINA. This method uses segmentations of multi-dimensional data to reconstruct noise-free MR images; these can then be used in Monte-Carlo experiments to measure the parameter stability of the segmentation, and also to assess the presence of most forms of potential bias.